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We study Kondo physics of a spin-1 impurity in electronic matter with strong spin-orbit interac¬ 
tion, which can be realized by depositing magnetic adatoms on the surface of a three-dimensional 
topological insulator. We show that magnetic properties of topological surface states and the very 
existence of Kondo screening strongly depend on details of the bulk material, and specifics of surface 
preparation encoded in time-reversal preserving boundary conditions for electronic wavefunctions. 
When this tunable Kondo effect occurs, the impurity spin is screened by purely orbital motion of 
surface electrons. This mechanism gives rise to a transverse magnetic response of the surface metal, 
and spin textures that can be used to experimentally probe signatures of a Kondo resonance. Our 
predictions are particularly relevant for STM measurements in PbTe-class crystalline topological 
insulators, but we also discuss implications for other classes of topological materials. 
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I. INTRODUCTION 

Recent explosion of interest in topological insulators 
(TispH is due in large part to the fact that they sup¬ 
port metallic states on their surfaces. The existence of 
these states (and hence the metallicity) results from the 
non-trivial topological nature of the Bloch wavefunctions 
in the conduction and valence bands of the bulk material, 
and is a robust feature. In contrast, the quantum num¬ 
bers associated with those surface states are not deter¬ 
mined by topology alone. Therefore they may vary from 
material to material, and depend on the surface prepara¬ 
tion. Understanding physical consequences of this non- 
universal behavior is one of the foci of our paper. 

A typical cartoon picture of surface states in a TI 
consists of spin-momentum-locked energy branches of a 
massless Dirac spectrum. This description cannot be uni¬ 
versally accurate. A crystal boundary breaks the inver¬ 
sion symmetry and gives rise to strong, rapidly varying 
in space, electric helds that define an effective surface 
potential for the electrons. Interplay between these field 
gradients and the bulk inter-atomic spin-orbit interac¬ 
tion (SOI), responsible for the non-trivial topological as¬ 
pects of these materials, renders this potential momen¬ 
tum and spin dependent. As we show below, this en¬ 
sures that measurable properties of the surface states 
cannot be determined by topological arguments alone. 
Details associated with a crystal surface can be accounted 
for via effective boundary conditions (BCs) for the elec¬ 
tron wavefunctiond^, and are completely excluded from 
the topological arguments involving only the bulk band 
structure. In this context. Ref. [5] argued that appro¬ 
priate BCs are essential for a sensible formulation of a 
bulk-boundary correspondence in TIs. Moreover, Ref. 
[3 pointed out a dependence of the spin texture of sur¬ 
face Dirac cones on the crystallographic orientation of 
the surface, even for simple BCs. 

In this paper we show that the spin behavior of surface 


states in three-dimensional (3D) TIs is highly sensitive to 
both the bulk band structure and surface properties. We 
consider semiconductors with different crystal symmetry: 
cubic lead chalcogenides (PbTe or PbSe) and tetragonal 
Bi 2 Se 3 -hke TIs, and demonstrate that magnetic probes 
(such as an external field or quantum impurities) can be 
used to efficiently differentiate between these two classes. 
Crucially, the sensitivity of TI surface states to the sur¬ 
face manipulation allows one to use TIs as a controllable 
environment for studying spin-dependent correlated phe¬ 
nomena in the presence of strong SOI. 

While some of the unusual magnetic phenomena that 
we argue for can be probed by measuring the response 
to a uniform magnetic held, in this paper we focus on 
the physics of a spin- ^ impurity deposited on the surface 
of a 3D TI. Kondo screening, whereby the impurity spin 
at low temperatures forms a singlet state with the Fermi 
sea, is one of the earliest and lucid examples of correlated 
many-body physic^ that remains rele vant in contexts 
ranging from heavy fermion system^^HSl ^;o nanoscienc^^. 
Advances in scanning tunneling microscopy (STM) al¬ 
lowed observation of this phenomenon on the atomic 
scal^i^®^, and granted access to manipulation of individ¬ 
ual Kondo resonanceJi^. Testing surface states of TIs via 
STIVP^ complements spin-polarize d ang le-resolved pho¬ 
toemission (ARPES) measurementJi^ES, and gives a di¬ 
rect probe of the Kondo effect. 

In its simplest form, Kondo screening involves only 
spin degrees of freedom of the conduction electrons. 
Hence it is sensitive to the spin-S'17(2) symmetry break¬ 
ing, provided in our case by the SOI. Previous works 
have shown that the Kondo effect survives in the pres¬ 
ence of spin-orbit scatterin^i^^, and weak (compared 
to the bandwidth) Rashba or Dresselhaus band SOP^HUl 
In some c ases, the latter can actually enhance the Kondo 
resonanc^^^^. The strong SOI regime is even more in¬ 
triguing. Indeed, the SOI can be viewed as a momentum- 
space magnetic “field” that aligns electron spins along a 
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particular direction (e.g. perpendicular to its momen¬ 
tum). When this field is large enough the spin degree 
of freedom of conduction electrons is effectively lost and 
cannot participate in the spin-flip scattering leading to 
the Kondo effec t. Ne vertheless there is substantial the¬ 
oretical evidenc^^mnHall indicating that a magnetic im¬ 
purity on a TI surface is screened by the surface metal. 
Remarkably the physical nature of this effect and spatial 
structure of the screening states have never been elu¬ 
cidated in the context of TIs. Understanding this phe¬ 
nomenon is also important from an experimental perspec¬ 
tive because magnetic probes (e.g. impurities or mag¬ 
netic field) coupled to surface electrons can be used to 
differentiate trivial and topologically non-trivial matter, 
providing an alternative to ARPES-based techniques. 

We demonstrate that the strong SOI leads to an un¬ 
conventional Kondo effect with an impurity spin screened 
by purely orbital motion of surface electrons. Specifically, 
we consider a simple band model of a 3D TI, and derive 
an effective Kondo Hamiltonian that governs the dynam¬ 
ics of the impurity spin at the TI boundary (that does 
not break time-reversal symmetry), taking into account 
the full 3D spatial dependence of surface-state wavefunc- 
tions. Because of the SOI this Kondo exchange has an 
XXZ structure and, in general, is strongly anisotropic. 
At low energies, the impurity spin forms a singlet state 
with the total electron angular momentum, and the sys¬ 
tem exhibits an emergent SU (2) symmetry, which is re¬ 
sponsible for the Kondo resonance. The SOI also gives 
rise to a transverse magnetic response when an external 
magnetic field applied normal to the surface results in 
an in-plane electron spin polarization, which may lead 
to interesting magneto-electric phenomena under driv¬ 
ing fields. This response is significantly stronger than 
an analogous effect on metallic surfaces with Rashba 
gopm. 

In Sec. [nj we describe our minimal model of a 3D TI 
and calculate its surface spectrum. Here we consider a 
continuum version of a lattice model studied in Ref. [HI 
Emphasis is put on the physical meaning of the quan¬ 
tum numbers involved in the effective description of elec¬ 
tronic states. In Sec. [m] we explain how both surface 
and bulk states play a role in determining the specific 
mathematical form of the relevant operators involved in 
the effective coupling between surface electrons of the TI 
and the magnetic impurity. Here, we contrast Bi 2 Se 3 
and PbTe-class materials. Section Hvl establishes the ef¬ 
fective XXZ Kondo Hamiltonian that governs coupling 
of these surface states to magnetic impurities, and ex¬ 
plains why this is a single-channel Kondo Hamiltonian 
despite its apparent two-channel form. In our approach 
we control the surface properties through BCs for elec¬ 
tronic wavefunction^ and show that surface manipula¬ 
tion provides an effective way of tuning parameters in 
the effective low-energy Kondo model and can be used 
to completely suppress the spin-flip terms and destabi¬ 
lize the Kondo effect. We study the physical proper¬ 
ties of the effective model and its unconventional Kondo 


physics in Sec. In particular, we investigate the trans¬ 
verse magnetic response to an external magnetic field and 
point to the resulting transverse spin textures as a dis¬ 
tinctive characteristic of the Kondo screening cloud in 
strong SOI materials. We also show how one can tune 
the Kondo effect and the characteristic temperature Tr- 
via surface manipulation. Our results can be directly 
verified in STM measurements in crystalline TIs like the 
lead-tin solid alloys Pbi_xSnxTe, but the above uncon¬ 
ventional Kondo physics can also be observed in well- 
studied Bi 2 Se 3 and BiSb. Finally, Sec. |VT] provides a 
summary and an outlook with questions that still re¬ 
main open. Two appendices with technical derivations 
complete the paper: Appendix [A| addresses the very im¬ 
portant problem of self-adjoint extensions of unbounded 
Hermitian operators, of key relevance to the analysis of 
bound surface states. Appendixj^exploits the axial sym¬ 
metry of the problem to construct surface states with 
well-defined total angular momentum. 


II. SIMPLE CONTINUUM MODEL FOR 
TOPOLOGICAL INSULATORS 

A. Model Hamiltonian 


To descr ibe electronic states in a TI we use Dimmock’s 
modeP^*^, defined by the modified Dirac Hamiltonian 

Hjo=v{a-p) + p(^X+^^^. ( 1 ) 

This effective Hamiltonian involves two spinful bands 
(conduction and valence) of opposite parity separated by 
an energy gap 2A and is written in terms of the 4x4 
Dirac matrices 

a = (cr^ (g) cr) = , /? = (ct^ (g) 1) = Q , 


with cr = denoting the usual Pauli matrices, 

and 1 - the unit 2x2 matrix. In Eq. Q the effective 
mass m* accounts for contributions from remote bands, 
and the velocity scale v is proportional to the momentum 
matrix element between conduction and valence Bloch 
states. In the following we shall adopt units with h= 1. 

The Dimmock Hamiltonian 0 provides a standard de¬ 
scription of electronic spectra in lead chalcogenides near 
one of the 8 equivalent L-points in the Brillouin zone. 
Note that despite the presence of SOI Em (^, is writ¬ 
ten in the basis of direct-product stated [Lg ) 0 |cr), 
where Lg denote spinor one-dimensional representations 
of corresponding to the conduction (Lq) and valence 
{Lq) bands, superscripts ± indicate spatial parity of the 
state, and tr = fj i is the electron spin quantum number. 
Eigenstates of Hj^ are 4-component envelope functions 
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which define the full electron wavefunction in the 
crystal: (a;|^') = Y.i=i, 2 \'^c,i{x){x\uc,i{kQ)) + 

ipy^i{x){x\uy^i{ka))]e’^°^ where (a;|M(c,„)y (fco))e*'“““ 
are Bloch states corresponding to band extrema at the 
point feo in the Brillouin zone {k ■ p method). The 
state does not need to have a definite spin quantum 
number due to the SOI usually present in TIs. In 
general, the indices i = 1,2 describe pseudospin states 
whose relation to the true spin will depend on the 
material. For instance, in PbSe-like systems the gap at 
the L-point is formed by non-degenerate representations 
of the single group D^d- The SOI does not affect these 
states besides shifting their energy, so the pseudospin 
states i = 1,2 can be identified with the eigenstates of 
[i.e. \a) = |t) or The (periodic part of the) 

Bloch basis functions can be taken as direct products of 
orbital and spin parts \uc,i{ko)) = |uc(/co)) O W)- 

The Hamiltonia n (ll| ) has a number of conserved “ten¬ 
sor” spin operatorsSSr Por us the important one is 

T = X p] = diag{[cr X p], -[(T X p]}, (2) 

with S = (l(g)(T). One can easily check that [H-d,T] — 0. 
Here we will only need = if3a^{a^ ■ p±), where “T” 
denotes xy vector components. The Dimmock Hamilto¬ 
nian 0 can be rewritten as 

Hj,=v{a^p, + iPa^T^)+(3(^A+^y (3) 

Note, that is a block-diagonal matrix whose elements 
are proportional to a “Rashba” SOI term [cr x p±\z- 
Effective mass models similar to 0 emerge in many 
narrow-gap semiconductors with strong SOP^, such as 
Bi 2 Se 3 . The relevant point in the Brillouin zone and the 
interpretation of the quantum numbers may differ de¬ 
pending on the material. For example, in Bi 2 Se 3 (sym¬ 
metry D^d at the F-point) the SOI is essential in deter¬ 
mining gap-forming stated, hence the basis states are no 
longer direct products. Even though there are still four 
states in the vicinity of the gap and the effective mass 
description is given by Eq. Q, the identification of the 
pseudospin with real spin (as for PbSe) is no longer pos¬ 
sible. These considerations are important for deriving 
an effective mass interaction Hamiltonian between the 
surface electrons and external probes such as magnetic 
field or magnetic impurities. Naturally, this interaction 
will depend on details of the bulk band structure of a 
material. Below we are going to illustrate this point by 
comparing the coupling of surface states and localized 
magnetic moments in lead and bismuth selenide com¬ 
pounds. 


B. Quasiparticle states in a half-space 

We are particularly interested in the localized surface 
states that form as a result of breaking translational in¬ 
variance. Consider a TI bounded by the surface z = 0 


(a) 



i9 ^ 



0 — Dimmock theory — ► Z 



FIG. 1. Panel (a) Geometry of the problem. The TI occu¬ 
pies half-space z ^ 0. The unit vector n is an outer normal 
to the surface. The red arrow represents the impurity spin, 
(b) Schematic k-dependence of the surface state wavefunction 

g , characterized by two length scales: small 1/A ~ IjnCv 
laded region) and large 1/q ~ a/A. The Dirac model 0 is 
valid at distances ^ 1/q. The impurity is located under the 
surface where the Dirac theory is applicable. The inset shows 
the dispersion of surface modes, (c) Stability diagram of sur¬ 
face states (lOl. Thick lines correspond to critical momenta 
Px = ±(A/ii) ctg d. In the white region no surface states can 
exist. In the (light) dark gray area, there are surface modes 
with (only one, r = — 1) both helicites. (d) Dispersion rela¬ 
tion ( |10| | for several values of i9. The upper (lower) branches 
(relative to the point p±_ = 0) correspond to r = =pl. The 
Fermi energy is ef =0. 


whose bulk states are described by Hd, see Fig. 0a). It 
follows that p± is conserved and, together with T*, can 
be used to classify quasiparticles states. An eigenfunc¬ 
tion V'piT of , T^ipp^T = Tp±ipp^r, has the form 


with 




( 4 ) 


where p± = |p_l| and e^^^^ = {px + ipy)/p±. The ampli¬ 
tudes a{z) and b{z) are determined by solving the remain¬ 
ing 2x2 boundary value problem. In Eq. (0) one can now 
replace with Tp±, hence reducing the number of inde¬ 
pendent Dirac matrices to two: /3 and a^. Their action 
on the z-dependent spinor part of ipp^xix) is equivalent 
to the action of and on a two-component wave- 
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function (a* b*y 


( ^(.^)^P±T 

[b{z)Up^^_r 

f 0,{z)Up^r \ 
\b{z)Upj^^-r j 


\_( h{z)Up^r \ 
) - \a{z)Up^^.,) 


( 0,{z)Up^r \ 

\-h{z)Up^,.,) 


-J>CT 
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a{z) 

b{z) 

a{z) 

b{z) 


which allows us to replace the Hamiltonian ([^ with a 
2x2 operator 


rr(2x2) 


= v{a^Pz — cr^Tpj_) + CT* 




(5) 


acting on two-component z-dependent wavefunctions. 
This reduction of dimension (from 4 to 2) is a direct 
consequence of conservation of . 

An important insight can be obtained by studying the 
simplest case of a hard boundary at z = 0 where the 
wavefunction vanishes, 'iplz^o — 0- Surface states with 
energy ep^^r = Tvp± exist for an inverted band structure 
when —m*u^/2 < A < Cpil. The surface-state ( unnor¬ 
malized) wavefunction is a coherent superposition of the 
conduction and valence bands 

J (6) 


and is characterized by 
dependent inverse length scales: 


two 



momentum- 
= m*v =F 


^ym*{m*v‘^+ 2A)+p±‘^. This surface state is stable 
only when q > 0, i.e. for p± ^ ^j2m*\A\ and merges 
into the scattering continuum for larger p^. For more 
complicated BCs, the problem of determining the sur¬ 
face spectrum from microscopic considerations is rather 
cumbersome (see Ref. Inland Appendix [A| for details). 

It is possible to simplify matters by considering the 
limit when m*v ^ P±- In this case vq « |A| 

and A « 2m*v ^ q. These lengths are illustrated in Fig. 
Bb). One can build a theorjEl valid on the scale ^ 1/? by 
neglecting 1/A. This small-p_L perturbative approach is 
similar to that used in hydrodynamics of weakly viscous 
fluidP^. In the bulk we can simply omit the /2m* term 
in Eq. Q, so the Hamiltonian takes the Dirac form: 


Hq = v{a.-p) + (iA. (7) 


Near the surface (at distances ~ 1/A) the situation is 
more complicated because the term p^ j2m* ^ A and 
cannot be neglected. Within this layer [shown in gray 
in Fig. [^b)] the electronic wavefunction varies rapidly 
in accordance with the BCs supplementing the Dimmock 
Hamiltonian Q. However, this complexity can be ab¬ 
sorbed into the BC for the Dirac Hamiltonian 0. This 
BC has to be consistent with the particle conservation, 
time-reversal and inversion (parity) symmetries, and can 
be written aP^ = 0 with 


( 8 ) 



FIG. 2. Panel (a) Helical structure of the surface state (101. 
Blue arrows indicate the expectation value of the electron 
spin which points perpendicular to the momentum and 
has a magnitude ~ sin-d. For •& = n, surface states carry no 
spin, (b) Schematic illustration of the Kondo interaction Hk 
in Eq. | |12[ ). At weak coupling one can ignore bulk-surface 
mixing induced by the impurity and assume that Hk ~ Hk ■ 
The impurity only couples to surface states (blue Dirac cone). 


and n - the outer normal to the surface. The bound¬ 
ary operator B includes one free parameter i? which ac¬ 
counts for microscopic properties of a realistic TI sur¬ 
face, and the behavior of the electronic wavefunction at 
the length scale ~ l/m*v. An exact connection between 
■d and the boundary conditions of the original fully mi¬ 
croscopic Hamiltonian is not unique in the effective long 
wavelength Dirac model. However, we show in Appendix 
[A] that to be self-adjoint in a half-space, the Hamilto¬ 
nian Q must have a single-parameter family of BCs. 
Consequently, variation of the parameter i? allows us to 
consider entire sets of possible surface properties real¬ 
ized in experiments. Physically, 'd controls the amount 
of particle-hole (p-h) asymmetry at the surface: The p-h 
symmetric case is recovered only when d = 0 or tt. The 
Dirac model Q is clearly less complete than the Dim- 
mock theory (IH, but it is much easier to work with. 

From now on we will focus on the problem defined by 
Eqs. § and ([^. Since'd can be chosen arbitrarily, we 
confine our analysis to the case A > 0 (no band inver¬ 
sion) and d G [0,7r]. Results for > tt can be obtained 
using charge conjugation ip —>■ a^ip*. The energy and 
wavefunction of surface states are given by 



epj_r = Tvp_\_ COS d — A sin d. 

(9) 

and 




— sind)t/p^i-\ (ip^-x^-qj, ^z) 

(10) 

Here M = 

y/qp^r/Ail — sind) and 



'Zpxr = —'rp.L sini? — (A/u) cosi? (11) 


is the localization wavevector and A - the area of the 
TI surface [xy-plane, see Eig. [^a)]. The stability region 
of the state ( [l0| is determined by the condition qp^r ^ 
0. For -d > ttJ2 the r = — 1 state exists for any value 
of p±, while the state with t = -|-1 is stable only for 
vp± < — Actgd. For d < 7r/2 the r = -|-1 mode is 


i? = 1 + /3sind -I- ■ n) cosd, 
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always unstable and the one with t = — 1 exists for vp± > 
Aotgr?. These regions are shown in Fig. Sc). The 
surface state enters the single-particle continuum at p± = 
p'J^ = ±(A/'!;) ctg-d. The function Cp^r is presented in 
Fig. Sd) for several values of'd. 

Surface states (10) are characterized by a helical 
spin distribution, shown in Fig. [^a), that depends 
on the BC^ as one can see by computing an ex¬ 
pectation value of the spin This average is 

(is) = [(1 — sind)^ cr-|-cos^ d cr^crtr^] Up^r = 

—gpj^T-Tsind (sin^pj^e^, — cos (pp^^By) ~ sini?, hence at a 
p-h symmetric point -d = tt, surface states (10) [and (^j 
carry no spin. This situation is quite different fr om the 
usual case of boundary-independent surface state^^. 

The single-particle scattering continua are defined by 
^p±Pz = ±\/r'^(P 2 +P_L^) + A^ with pz ^ 0. Note that 
^p±Pz is doubly degenerate w.r.t. r. The corresponding 
(unnormalized) wavefunction is 


'^P 


-(a;) 


(A- 


J sin K Up 


—iu [pz cos K + Tp± sin k] Up^ 


where k = PzZ + C and 


tgC = 


vpz COS d 


(1 -I- sind)(A -I- Cp^pJ — Tvp± cosd ’ 


Finally, we make two general remarks. First, for d = tt 
the Dirac surface state ( [l0|) has a structure similar to 
its Dimmock counterpart™ . An additional negative 
sign in the lower component of the spinor in Eq. (10) 
appears because in the Dimmock theory (Q we used A< 
0, while in the Dirac Hamiltonian ([^ A > 0. In the latter 
case the sign of A can be flipped by a unitary rotation 
Hq —>■ HqA and B —J. with A = (cr^ 0 1). After 

this transformation the Dirac surface state wavefunction 
(fTo|) at d = TT becomes identical to Eq. ([^. Hence, 


conclusions obtained using the Hamiltonian ([^ should 
also be applicable to the Dimmock model. 

Second, one has to prove that the Hamiltonian 0 is 
self-adjoint in the space of wavefunctions satisfying the 
BC (|^, which is necessary to guarantee that the Dirac 
model isphysical and our conclusions can be linked to 
experimentally observable quantities. In Appendix [A| we 
show that this is indeed the case and the BC (|^ defines 
a self-adjoint extension of the Dirac Hamiltonian ([^. 


(and has units of energy x volume). Based on contribu¬ 
tions from different parts of the electron spectrum, the 
operator Hj^ can be decomposed as 

Hk = 'PsHk'Ps + VbHKVb + {VsHxVb + h.c.). 

Here Vs (Vb) is the projector on the surface (bulk) sub¬ 
space with Vs+Vb = 1 ■ The first two terms have matrix 
elements only between surface and bulk states respec¬ 
tively, the last term describes surface-bulk mixing in¬ 
duced by the impurity. Since bulk states are gapped, the 
pure bulk contribution cannot support Kondo screening 
(due to a vanishing density of states at the Fermi surface) 
and can be omitted. For Jk/^c ^ A, with the “Comp¬ 
ton” length scale Ic = v/A, the off-diagonal surface-bulk 
mixing term is perturbative, and can be neglected in a 
zeroth order approximation [see Fig. [^b)]. In the follow¬ 
ing we will focus on the surface term, = VsHkVs- 
As already mentioned in Sec. H A[ the relation between 
real electron spin and the pseudospin index in the Dirac 
Hamiltonian is material-dependent. We will consider the 
simplest case of PbSe-class materials where the electron 
spin operator coincides with the pseudospin and has the 
form 


s(®o) = ^c1'(a:o)Sc(a;o) = ict(a;o) fg ® 


c(a;o), 


where c{x) is the annihilation operator that corre¬ 
sponds to the quasiparticle eigenstates (10), c{x) = 
'>Pp±T{x)cp^r + bulk modes. 

The surface part Hjp is obtained by computing matrix 
elements of S between states (10): 




2 V'pyr' 

X Up, [(1 — sin 'd)‘^cr + cos^ Up^r = 

^ ■\JQp'j^r'Qp±T Up/^^, [ sind -t -cj Bz\Up^.zi 

where cr^ = + <7^6^ and we used the identity 

Up^,-r — a^Up^r- The full effective Hamiltonian is: 


Fief = iFo + = I] e 


''P±tc\)^t^P±t 


+ ^S-s,, (13) 




III. COUPLING THE TOPOLOGICAL 
INSULATOR TO SURFACE MAGNETIC 
IMPURITIES 

The Kondo Hamiltonian that describes the interaction 
of the two electronic bands with an impurity on the sur¬ 
face at a; = a;o = (0, 0,0) has the form: 

Hk = JkS ■ s{xo), (12) 

where S is the impurity spin and s(a;o) is the electron 
spin density at Xq, the coupling constant Jk is positive 


with = Vss(xq)Vs- 

Sc = X! + (^""ezlUp^rCp^T 

pW 

P±r 

/ / 

and Qp^t = y/(lp'^T'(lp±T- For d = tt (hard wall BCs in 
the Dimmock model) the coupling of electrons to the im¬ 
purity spin is purely Ising-type. Since the impurity spin 
cannot be dynamically flipped, there is no Kondo effect 
in this p-h symmetric case. This offers the possibility 
to control the Kondo screening by surface manipulation 
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via the boundary parameter i9. Even though the bulk 
Kondo coupling (12) is 5'C/(2)-symmetric, Eq. (13) de¬ 
scribes a Kondo impurity model with an XXZ exchange 
anisotropy, which is a direct consequence of the inversion 
symmetry breaking at the surface. 


Due to factors the Hamiltonian (13) is equiva¬ 

lent to a Kondo model with spatially non-local exchange 
couplings. This can be seen by rewriting Sc hr terms of 


the fermions Cp^^a = J2T(^p±'r)a^p±r with a = f, J,: 

Sc = ^ Ma>p'{p'j^)[-sm'da-^+ 

P'±P± 

+ Cr l3Mpa{p±_)Cp,^^,Cpj^a: 

where Maf3{p±) = 'Er V^pI^iUp^T)UUp^r)p = 
Qo^a0 “f Q zip'a 13 ^ PJ-Iz/PI. with Qq — 2 \/ 

and Q 2 = i Et '^y/lp±r- When r? = tt — for small 
|Jd| ^ TT and ^ 1, to the lowest order we have 

qJ ~and Qo « [l " (f )^(l + w)]. and 


p\p± 


— {-eras'd + e^a ) + —- P±) • cr+ 

V 2 


+ +PJ.) X + Kp± - 


with 2Fp>^p^ = 1 -|- (^)^(p± +P±)^- The first term in 
this expression will give rise to the usual (local) Kondo 
interaction. The third term describes a purely orbital 
mechanism to flip the impurity spin via a non-local p- 
wave coupling with the conduction electrons. Einally, the 
longitudinal terms (proportional to e^) reflect an effective 
Zeeman field originating from electron in-plane motion. 


Because Urp^ are eigenstates of [cr x Pa_\zi Hei in Eq. 


(13) describes a two-dimensional system of electrons sub¬ 
jected to a Rashba SOI and interacting with a magnetic 
impurity. Erom Fig. 0d) it follows that by tuning {) we 
can make one chirality r almost completely disappear, 
which is equivalent to having a strong SOI dominating 
single-electron kinetic energy. 


The effective m odel ([l^ seems to be incompatible 
with recent resultj^ l'^^ l '^^ arguing that there is always 
Kondo screening at the surface of a TI. The root of 
this discrepancy is the common assumption that TI sur¬ 
face states can be considered as helical Dirac (or Weyl) 
fermions. From Eq. (§, the effective single-particle sur¬ 
face Hamiltonian has the form i?Q®' = Up^T^p^rU^j^^ = 
?;cosi?[(txp_l ]2 — Asind, and the usual case encountered 
in the literature, TJq®' = —v[cr x pEzi is recovered when 
•d = TT. The above assumption is not universal: While 
the free particle dispersion relation is captured correctly 
by it is non-trivial to couple these surface electrons 
to external probes, e.g. impurities or an external mag¬ 
netic field. Interaction terms involving TI surface states 
have to be derived carefully taking into account bulk and 
surface properties, and are material dependent. 


Indeed, our results will be completely different for 
Bi 2 Se 3 . The tetragonal band structure of this material 
dictates that an effective mass expression for the elec¬ 


A{Sdf 




p±p± 


4A 


[P± X P±] 


r' . c 


trons spin iP^ 

s'(a=o) = ^c^ixo)'S'c{xo) = [xq) ^^o)- 


The cancellation in the spin matrix element which led 


to the factor sin d in Eq. (13) does not occur and we 


recover an isotropic (XXX) Kondo Hamiltonian whose 
structure is essentially independent of d: 


Hli = -J^Q^pEr 


p±r^p±T • 


In the particular case of d = tt (when Q = const) it 
is indeed admissible to use the Dirac-Weyl description 
of surface states with the Pauli matrices in the effective 
Hamiltonian being the true electron spin, as described 
for example in Ref. [TJ However, as indicated above, for 
PbSe-class materials this is not the case. 

Another way to experimentally distinguish the above 
two classes of materials is by their response to an ex¬ 
ternal homogeneous magnetic field h applied parallel to 
the surface. Without loss of generality we assume that 
h — he^. The surface electrons couple to this field via a 
Zeeman term Hz = — | Ei'^sC^{xi)k^c[xj)'Ps\^^Q with 
A = E or S'. In Bi 2 Se 3 -like TIs with d = tt, the full 
single-particle Hamiltonian is iJo = ~Ep^ x 

Pa.\z + olEp^P' Hence, the only effect of h on 

surface states is to shift the Dirac cone in the Brillouin 
zon^^. On the contrary, for = tt surface electrons in 
lead chalcogenides do not couple to the transverse field 
at all, because s“ = 0. The Zeeman coupling appears 
only to the order (tt — d)^. For d ^ tt, the Dirac-Weyl 
description of surface states in terms of idg®* is meaning¬ 
less, regardless of the material. 
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IV. EFFECTIVE SURFACE HAMILTONIAN: 
ORBITAL NATURE OF SCREENING 


To gain insight into the physical properties of the 
Kondo impurity model (13) we will exploit its axial sym¬ 


metry which guarantees conservation of the z-component 
of the total angular momentum jz = -I- with Iz be¬ 

ing the orbital part. The fermions Cp^r can be expanded 
in the angular momentum basis: 


r — f. . p 

^p±T — / j ^p±mT'i ^p±mT 

m 




where the integer m S (— 00 , 00 ). The sum over </). 
has to be understood as 


277 


277 


P± 

We also 

dehne a sum over the radial momentum p±: 

so that Moreover, 

. Using these relations one can 
show that Cp^rriT satisfy the fermionic anticommutation 


relations {cL 




} — dp'^p mdr'T • 


The fermion spin density in Eq. (13) becomes: 


=isini? ^ Qpir 


Or'''PrlT’ 


P±T 


2 

p±T 


Qp±t ’’'‘"p^1tTpj_1t)- 


Because only m = 0 (s-wave) and to = 1 = — 1 (p-wave) 
angular harmonics enter these expressions, we can define 
new fermionic degrees of freedonP^ 

^P±T'[ — Cp^Or, ^pxT^ — (f4) 

These operators create surface electrons with total angu¬ 
lar momentum jz = ±1/2 (see also Appendix [b|. 

To give microscopic meaning to the operators ([T^, it 
is instructive to compute the local electron spin density 
at the surface that corresponds to a state with one a- 
particle, i.e. an expectation value in the state |lp^Tp) = 
®PiTp|0) of the operator 


s(a:j_) — 2 'y ' 

p'±p± 

p'p 


V, 

0=0 


wher e jjp , mr = V'pj_m(/ 0 )r is the surface state wavefunc- 
tion (101 in the angular momentum basis (cf. Appendix 
[b| , r?r = to(p'), and TO(t) = 0 and to( 4,) = —1. In the 
polar coordinates a;j_ = (rcosp,rsinip), we have 


(Ipi T/i |s(a?j_)|l 


\ _ 


(15) 


x<{ -Tsm'dJoip)Mp) er + -[Jo(p) - ^1 (p)] e. 



FIG. 3. Spin distribution {lp^rp|s(»_L)|lp^pp) = {s(r)), Eq. 
( |15[ ) along radial direction with r = 1 and ■& = O.Ott. (s’')[(s^)] 
is the radial [z] component. The blue line is the “total” spin 
s*°*^ = y/ (s”)^ -I- (s^)^. Blue arrows show schematic spin dis¬ 
tributions at a fixed radius r. Thick red dots indicate the 
impurity location at the origin xo = (0,0,0). 


with Br = (cosp, sin(p), p = p±r, and Jnix) is the n- 
th Bessel function of the first kind. The upper (lower) 
sign corresponds to /i = f (i)- The spin distribution 
( [I^ is shown in Fig. Unlike the plane-wave states 
c^p|0), the wavefunctions carry no net spin, 

i.e. J d X_}_ (Ip I rMl^ (^-L )|fpi r/r) 0- 


Using operators (14), we can rewrite Eq. (13) as 




■Pr 


(16) 


Jk 




+ o'.,i,,ez)a 


pi 


p^T 

P±r 


Here we assumed implicit summation over pseudospin 
indices /i and p' = f,omitted all angular harmonics 
TO 0,1 which do not couple to the impurity, and dis¬ 
regarded the negative sign in the W-term. This sign is 
irrelevant and can be switched by a unitary transforma¬ 
tion Hef —WUefU with U = 25'*. Elementary spin-flip 
scattering processes in Eq. correspond to dynamical 
mixing of the spin distributions (dH) and are schemati¬ 
cally illustrated in Fig. B^) [and should be contrasted 
with spin-flip scattering in the usual metal without SOI 
depicted in Fig. B^)]- 

The Hamiltonian ([T^ appears to describe a magnetic 
impurity coupled to two conduction bands (channels) la¬ 
beled by the helicity index r = ±1. However, this is not 
actually the case as can be easily demonstrated by con¬ 
verting Hef to the energy representation. We shall con¬ 
sider only energies within the bandgap, —A ^ e ^ A and 
assume that tt/ 2 < d ^ tt, so cos d ^ 0. There is a one-to- 
one correspondence between r and energy (i.e. helicity of 















TI surface 


no SOI 


the state and its energy in the upper or lower Dirac cone). 
From Eqs. ([^ and @it follows that r = +1(—1) cor¬ 
responds to energies e < —Asin-d (e > —Asind). Since 
Qp^T and depend only on the product Tp±, 


A -I- e sin 
u| cos'd] 


(17) 


Notice that q{e) ^ 0 for all e within the gap. Next we 
derive the density of states (DOS) 5r(e)- For r = -|-1 
one has ^ = I- a sin ,9 -h Asind)/27ru^ cos^ d = 

"“’’de g+(e). Similarly for r = -1: = 

f-Asini}^’^ = (e-H Asin'd)/27ru2cos2d. 

Hence for all energies 


fl(e) 


|e -I- Asind| 
27ru^ cos^ d 


(18) 


Finally, we introduce new operators = 

0'P±{e)Tp/ V 5 (e) with anticommutation relations 

Oe/j} = Sp>pS{e' — e) which allow us to reduce idef 
to a single-channel form 





This reduction from a two-channel form (16) occurs be¬ 
cause of the unique correspondence between energy and 
helicity peculiar to surface states. 


V. UNCONVENTIONAL KONDO PHYSICS 


The Hamiltonian (16) describes a Hondo impurity 
model with an anisotropic (XXZ) exchange coupling and 
a DOS (18) that can vanish at the Fermi level, e = 0, if 


the BC d = TT is satisfied. In this limit two effects si¬ 
multaneously ensure that the Hondo screening does not 
occur, and the impurity spin effectively decouples from 
the surface metal. First, from numerical renormaliza¬ 
tion group calculations, for linearly vanishing DOS and 
particle-hole symmetry the critical Hondo coupling does 
not exislP^. It is worth noting that this behavior is not 
captured by the standard mean field theorie^. Second, 
in our system the spin-flip scattering is proportional to 
sind and therefore disappears at d = tt. This effect is 
already present at the mean field level. Hence the decou¬ 
pling of the impurity from the metallic surface states at 
d = TT is inexorably linked to the anisotropy of the spin 
scattering stemming from the bulk band structure. 


For f < d < TT there is a finite DOS (18) at the 


Fermi surface, and for temperature T below a character¬ 
istic Hondo scale Tk, the impurity spin is screenecpESl 
This Hondo effect occurs due to the orbital motion of 
conduction electrons [Fig. |^a)] unlike the conventional 
case when the impurity is screened only by itinerant spins 


(a) 


(b) 




FIG. 4. Panel (a) Upper plot: Spin-flip scattering processes 
leading to the Hondo effect. The red [blue] arrows denote im¬ 
purity spin at = 0 [local spin ( |15| l in the conduction band 
at a distance jccxj = r from the origin]. The impurity spin is 
screened by the orbital degrees of freedom (coupled flips of the 
electron spin and orbital angular momentum G). The lower 
plot shows a spiral spin structure ( |15[ ) along a radial direction 
(p = P±r) away from the impurity, (b) Same as in panel (a) 
but for a conventional metal without SOI. Only conduction 
electrons in the s-wave state couple to the impurity and the 
orbital angular momentum does not participate in the Hondo 
screening. The spin direction (along an arbitrary direction n) 
does not depend on the radial position. 


[Fig. Qb)]. More precisely, the impurity spin forms a sin¬ 
glet with the total angular momentum j of the surface 
states. This unconventional mechanism for the Hondo 
screening originates from the strong SOI that couples 
spin and orbital momentum of electrons in TIs. 

In the following we would like to address the physi¬ 
cal manifestations of this unconventional Hondo effect. 
We first demonstrate the appearance of a transverse spin 
linear response to a longitudinal external magnetic field. 
We next consider the effect of temperature and study the 
dependence of the Hondo temperature on the electronic 
surface properties parameterized by t). Although we fo¬ 
cus on the model (16) obtained in the context of TIs, 
results of the present section are applicable to Hondo 
physics in any two-dimensional metal with SOI. 


A. Transverse local magnetic response 


The simplest manifestation of the spin-orbital nature 
of the Hondo effect on a TI surface can be found in 
the zero temperature (T = 0) linear response to a weak 
magnetic field h acting on the impurity. Assuming that 
h = liEz points perpendicular to the surface, the field 
correction to the model (16) is 


Hmag = -hs^. 
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According to Eq. (15), surface states with /r = f and 
4 , correspond to different (opposite) radial spin distribu¬ 
tions. In the Kondo singlet state both configurations are 
equally probable and the total spin in the a; 2 /-plane van¬ 
ishes. However, in an applied magnetic field the impurity 
spin is weakly polarized creating a population imbalance 
of electrons with different /i’s. This imbalance results in 
a transverse local (i.e. at a fixed distance from the im¬ 
purity) spin polarization in the conduction hand, see Fig. 
|5];a). 

To calculate the field-induced transverse magnetiza¬ 
tion we use the standard variational approaclP^ for the 
Kondo problem, and assume that i? < tt so that all 
T = -1-1 states are fi led and the Fermi level lies in the 
T = — 1 cone in Fig. lid). At weak coupling ^ 1 

one needs to keep only r = — 1 terms in Eq. (16), hence 
in the rest of this Subsection we will omit r in the sub¬ 
scripts. The variational wavefunction has the forirP^ 


1^0)= XI ® 1“)’ (19) 


where kp = 


Ae 


is the Fermi momentum, and |FS) 


v\ COS 

and I a) are the Fermi sea and impurity spin states 
{a = t) i) respectively. There is an implicit summa¬ 
tion over spin indices. The two terms in (19) cor¬ 


respond to singlet (y'*) and triplet (x*) components 
with XpQ = T The latter satisfy the re¬ 

lations XuaXua = xLxL = XuaXL = 0, and 


flCt A. fj,ct AficxAfiCk 

SluXlfi = ^Xpc- The state 


fiCkAfj,oc 

19|) is normalized accord¬ 


ing to (V'olV'o) = E 


p±^kF (|A 

P± 




p± I 


= 1 . 


The amplitudes Ap^ and Bp^^ are variational param¬ 
eters determined by minimizing the functional JF = 
(V'ol-ffef + I?mag|V'o) “ (Aps “ A) ('00 IV'o) , where A is the 
Lagrange multiplier that plays the role of an energy shift 
due to the Kondo screening. To the first order in h, a 
straightforward calculation yields 


Ap 


Jic(l + 2sini?) 


4A 


_ i + A 

(cpx + A)^ V 


with p_L > kp and c = Y^p^^kp V^Ap^. The eigen¬ 
value A is determined from the non-linear equation 


^ Tpc (1 T 2 sin id ) 


dpi 


4A 


-r—, ip± + A 

Pr^kp 


( 20 ) 


At weak coupling the sum can be computed as 
X Ep_L • • • = C de « <?(0)5(0) In f [g(e) 

is given in Eq. 0] which means that In ^ « 

—Sttu^I cos^ id]/A'^J k sini?(l-|-2 sini?). Then, the normal¬ 
ization constant c is given by [ j^(i4sintf) ] "■ 

If, as is commonly done, one identifies the energy 
shift A with the Kondo temperature Tp, we find that, 
as > TT, Tp vanishes exponentially as Tp ~ 

A exp[—S ttu^/A^ J/f (tt — d)]. Note, however, that in this 


(a) 



(b) 



FIG. 5. Panel (a) Schematic plot of the transverse magnetic 
response on a TI surface. A magnetic field h applied normal 
to the surface causes a radial electron spin polarization, (b) 
Same as panel (a), but for the Kondo effect in a usual two- 
dimensional metal without SOI. There is only longitudinal 
magnetic response. 


approach for a finite DOS at the Fermi level the vari¬ 
ational energy shift does not vanish when spin-flip pro¬ 
cesses are suppressed. Consequently, in next subsection 
we take this effect into account and define the Kondo 
temperature using the slave-boson method. 

The field-induced transverse spin distribution in the 
ground state |0o) is straightforwardly obtained using Eq. 


(B2) and the discussion in Sec. IV 


(0o|s’'(a;_L)|0o) = 


sin id 
~A~‘ 


^ X! \/ dp'p dp± A. 

p'tPp^kp 


p'^Bp^Jnip'j_r)Jmip±r), 


with n,m = 0 and 1. With the aid of the above expres¬ 
sions for A, c, Ap^ and Bp ^, we finally arrive at 


( 0 o|s’'(a;j_)| 0 o) = - 


Ah sin d 


(1 -I- 2s\n'd)Jp 


Jo{kpr)Ji{kpr), 


where we also employed a weak-coupling approxima¬ 
tion for the energy integrals def{e)/{e + A)" « 

/(O) dej (e -I- A)" (/ is a smooth function and n ^ 0 is 

an integer). Due to the structure of the variational state 
the spin distribution is identical up to a prefactor to 
Eq. (15) with p_L = kp [see also Fig. |^. 

The transverse magnetic response, i.e. nonzero lAYUA 
h, can be viewed as a variation of the Edelstein effeclP^I^ 
an applied magnetic field creates an imbalance of differ¬ 
ent orbital angular momentum states that couple to the 
impurity, which in turn induces a radial spin polarization. 
This phenomenon exists only due to the SOI and is ab¬ 
sent in metals without SOI [see Fig. [^b)]. Therefore, by 
studying the spatial structure of the Kondo resonance, 
for example by spin-polarized STM, one can differenti¬ 
ate between topologically non-trivial and trivial states of 
matter. Field-induced radial spin spirals similar to Figs. 
Sa) and[^a) were reported in Ref. [321 in connection to 
Kondo screening of magnetic impurities on gold surfaces 
with a weak Rashba SOI ap. In that work, the transverse 
susceptibility k± = {s^)/h ~ ap. Our results deal with 
an opposite limit of strong SOI and hence k±_ depends 
only on Jp and the boundary parameter d. 
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In the absence of an external field h, the ground state 
wavefunction (19) is an S'17(2)-singlet, despite the XXZ 
anisotropy of the Kondo model (16). This is an exam¬ 
ple of the general irrelevance of exchange anisotropies for 
the Kondo physicJ^^. However, in our case this emergent 
SU{2) symmetry is quite non-trivial because the impu¬ 
rity spin forms a singlet with the total angular momentum 
of the surface electrons [see Fig. |^a)]. Coupling to the 
orbital motion ensures that this singlet-formation is the 
physical mechanism responsible for the Kondo resonance 
even when electron spins are quenched by the strong SOI. 


B. Slave-boson mean-fleld approach 

In the previous Subsection we assumed that for any 
■& < TT the impurity is screened by surface electrons with 
only one helicity r = — 1. Here we verify this conjecture 
by studying the mo del in Eq. (16) within the slave boson 
mean-field approaclP^. This analysis also provides an 
extension of our previous results to finite temperature. 

First we introduce a pseudofermion representation 
of the local spin S = with the constraint 

fjifn ~ language the interaction term in 

Hg{ can be written in a compact form 


Hef=- 


Jk sindxlxo + Jk 


1 — sin d + 

- 2 -'X-L- 


7 l-2sin^? ^ 

Jk . 2_^ 


P±‘t^ 


( 21 ) 


p±-^ 

P±T 


The slave bosons are 




defined ai 
with I = 


la 


: xi = 

0,1,2,3 and 


= [see Fig. [^. Notice, that the zero energy 
in Eq. (21) is chosen such that it eliminates Xzy which 




is necessary since energies of the states with condensed 
Xo E^nd Xz bosons (see below) are only different when 
spin-flip scattering is present. This procedure adds a 
potential scattering term that preserves the impurity 
spin and is therefore irrelevant for the Kondo physics [cf 
Ref. 133]. 

The mean-field appoximation amounts to treating the 
pseudofermion constraint on the average via a chemical 
potential Ef, and assuming that the ground state corre¬ 
sponds to condensation of the xo boson, i.e. (x_l) = 0 
but (xo) = Xo 0- The mean-field Hamiltonian, 




= E 


it 




MF — / , '^p±T^p^rp'^P±rfi 

P±Tfl p 

JifSini? ^ 


2y4 


P±'rn 



FIG. 6. Kondo temperature, computed from the Nagaoka- 
Suhl equation (22), as a function of the BC angle [see Eq 
(|^]. The arrow shows increasing values of the dimensionless 
Kondo coupling ok = — 0.05, 0.1, 0.2, 0.4, 0.7, 

1.0. Inset: Energies involved in the Hamiltonian (211. The 
blue color indicates the condensed slave boson xo- 


((SrrAi;aplr>))> ((//r;/,!))> and ((/^iot^^p)). A direct 
calculation yields: 


(a -al ) = 


Zp^T 


Jk sini? XOi/^Pi-r 

\/ 2 A oj ^p±T 


ifp,y^p±Tp,)<^ ~ 


iffJ.: ^p'^t' p)^ 

J/f sin'dxov^?)^ 


- €p^r) [to + Ef - H(a;)] ’ 

where we introduced the Fourier transform {{A; B)) = 
^ and the impurity self¬ 

energy 


[J^sin^xop y gpvr 


^P±T 


The mixed Green function {f^; allows us to con¬ 

struct the self-consistency equation for xo^ 


Xo = 


1 


E! V Qp±r{0‘p^Ttifp) — 

P±rp. 


1 


E! y/Qp±r 

P±Tp 


doj Ap^rp (a^); 


with the spectral function 


can be diagonalized using the equati ons o f mo¬ 
tion method for retarted Green function^^^Ull which, 
for fermions, are defined as {{A; B)) = —i9{t — 

t'){{A{t), B{t')}) [9{x) is the Heaviside step func¬ 
tion]. We will need three types of Green functions: 


4 / \ i if^p±Tp){fp^ ^p^Tp}^—io 

Ap^rpiuj) = — e-/^ + 1 ■ 

At the Kondo temperature Tr-, defined by Ef{TK) = 
Q,{Tk) = 0, the above self-consistency condition reduces 
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to the Nagaoka-Suhl equation 
^^_ Jk Sind pv.^. 


A 


Pj_t 


[e'^Pi 


r/TK 


- 1 ) 


( 22 ) 


where P.V. indicates the Cauchy principal value. This 
expression generalizes the Yosida equation (20) for the 


case where both helicities r are allowed to participate in 
the Kondo screening. 

For d sufficiently distinct from tt and weak coupling 
we only need to consider conduction band states near 
the Fermi energy. All of them have the same helicity 
due to the one-to-one correspondence between r = ±1 


and energy leading to the helicity-independent DOS (18). 


This fact justifies our assumptions made in the previous 
Subsection. 


Beyond weak coupling, the sum in (22) can be com¬ 
puted numerically using Eqs. 0 and (|18|) for q{e) and 
g{e). The dependence of the Kondo temperature on d 
is shown in Fig. Asymptotically, for — ?> tt, the 
Kondo temperature Tk ^ Aexp[—27ru^/A^ Jif(7r — d)^] 
is exponentially suppressed, albeit its functional behav¬ 
ior is different from that obtained using the variational 
approach. Since at d = tt, the XY term in (161 vanishes, 


there is no critical Kondo coupling that would yield a 
finite Tk at this point [cf. Ref. Wf\ . 


VI. DISCUSSION 

In the present work we advocated the use of magnetic 
probes to test and tune the unconventional phenomena 
at topological insulator surfaces. We showed that phys¬ 
ical characteristics and quantum numbers of the surface 
states are quite sensitive to surface properties encoded 
in boundary conditions for electron wavefunctions, as 
well as the structure of bulk Bloch bands. Moreover, 
we demonstrated how the combination of spin-orbit in¬ 
teraction and non-trivial boundary conditions leads to 
an unconventional Kondo screening of dilute magnetic 
impurities on the surface of a 3D topological insulator. 

We considered a localized spin S = ^ (magnetic) im¬ 
purity atom deposited on the (111) surface of a PbTe- 
class narrow-band semiconductor, and derived a low- 
energy effective theory that governs the coupling of this 
local spin to surface electrons taking into account the 
full 3D structure of surface-state wavefunctions. The re¬ 
sulting Kondo impurity model is spatially non-local and 
anisotropic [AAZ-like, see Eq. (16)]. Interestingly, both 
of these features are controlled by parameters defined 
by the boundary conditions, in our case d, that deter¬ 
mine the magnitude of the particle-hole asymmetry at 
the surface [see Eig. 1(d)]. Specifically, at the particle- 
hole symmetric point d = tt the AY component of the 
Kondo exchange interaction vanishes, signalling an insta¬ 
bility of the Kondo screened ground state (for any amount 
of surface gating) due to the lack of spin-flip processes. 

When the particle-hole symmetry is broken by the 
boundary conditions, we find that the impurity spin is 


fully screene d by the surface electrons, in agreement with 
earlier work^^lESIMIlII However, unlike the conventional 
Kondo effeclP, here the local spin forms a singlet with 
the total angular momentum of itinerant electrons (as op¬ 
posed to only their spin) and is screened mainly by the 
orbital electronic degrees of freedom. This effect origi¬ 
nates in the strong spin-orbit interaction that underpins 
the helical structure of the surface states, and manifests 
itself in a transverse spin response: A weak, normal to 
the surface, magnetic field induces an in-plane electron 
spin polarization [see Eig. [^a)] which locally resembles a 
q = I magnetic vortex (see Ref. I55|l with itinerant spins 
aligning along the radial direction. 

The sensitivity of the Kondo screening to specific sur¬ 
face properties shows that it is impossible to provide a 
universal theory of topological insulator surface states 
based solely on topological argument^^ without involv¬ 
ing knowledge of the boundary conditions for the Bloch 
states (see Sec. [Tljand Ref. 0), and, as elaborated in the 
present work, the specific bulk band structure. Most im¬ 
portantly, the latter defines the set of relevant effective 
operators that parameterize the surface theory. Indeed, 


in Sec. Ill we demonstrated that for a Bi 2 Se 3 -like tetrag¬ 
onal material the form of the surface Kondo interaction is 
completely different (isotropic, AAA-like) than in cubic 
PbTe-like systems (anisotropic, AAY-like). 

This physical non-universality of topological surface 
states can be exploited in experimental studies of topo¬ 
logical insulators, for instance to control the surface spin 
polarization with external electric and magnetic helds. 
Although we focused on magnetic impurities, our analysis 
can be generalized to any magnetic interaction, e.g. the 
Zeeman coupling of surface electrons to external fields. 
Eor Bi 2 Se 3 -like materials, the only effect of an in-plane 
magnetic field is to shift (neglecting the Eermi surface 
warping) the Dirac cone in the Brillouin zon^^. How¬ 
ever, in PbTe-like crystals with a particle-hole symmetric 
boundary {d = tt) such field does not couple to surface 
states at all. In general, this coupling can be tuned by 
surface manipulation. The above result shows a con¬ 
venient way of discriminating between different types 
of topological insulators by using interactions of surface 
states with external magnetic probes. 

The transverse spin structures in Pig. [^a) can be ob¬ 
served in scanning tunneling microscopy measurements of 
the local spin-polarized density of states around the im¬ 
purity, or nuclear magnetic resonance experiments. This 
predicted effect is not peculiar to topological insulators 
and should in fact exist in any strong spin-orbit coupled 
metallic host. A similar idea of probing the local spin 
polarization around magnetic impurities in a metal with¬ 
out spin-orbit interaction, i.e. the analysis of the Kondo 
screening cloud, was discussed before^. Unlike our anal¬ 
ysis, in that work the magnetic field induced only a longi¬ 
tudinal (and no transverse) spin polarization [Pig. [^b)]. 

Pinally, we comment on the role of impurity charge 
fluctuations in multiband Dirac-like materials with 
strong spin-orbit coupling. The standard Kondo impu- 
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rity Hamiltonian is typically derived from the Anderson 
impurity model via a Schrieffer-Wolff (SW) transforma¬ 
tion assuming that charge fluctuations at the impurity 
get suppressecP. In the absence of spin-orbit interaction, 
the virtual transitions included in the SW transforma¬ 
tion preserve the electron spin quantum number. The 
effective Kondo exchange then depends on momentum 
only via energy, and near the Fermi level can be approxi¬ 
mated by a constant value. This situation may change in 
a spin-orbit coupled system when the electron transitions 
between local and itinerant states are accompanied by a 
spin-flip. For a PbTe-like host these processes can be 
captured with a modified 3D Anderson impurity model 

^^AIM = + ^ [K^/3(p)CpQd^ + V'a^(p)/lpQ,d^-|-h.C.] , 

written in terms of the fermion operators dj,, and 
that create electrons in the impurity orbital, conduction 
and valence band, respectively. Hq = Hu + Hd, with Hu 
from Eq. and Hd representing the self-energy of the 
localized electrons. The matrix amplitudes and 
describe hybridizations of the local impurity level with 
electrons in the conduction and valence bands. 

A phenomenological form of these amplitudes can be 
obtained from general symmetry considerations. We re¬ 
quire that Haim has the same symmetries as the non¬ 
interacting Dimmock model Hu, in particular, time- 
reversal invariance and symmetry w.r.t. spatial inver¬ 
sion V. The former demands that and F” con¬ 
tain spin (via the Pauli matrices) and momentum in 
even power combinations, e.g. or cr ■ p. The inver¬ 
sion symmetry dictates which of these terms actually 
occur in each hybridization amplitude. Under V, local 

V 

fermions are invariant da —t da, while conduction and 
valence band electrons transform a^cp„ ^ C-pa and 

■p 

hpa —^ —h-pa- This means that (F’') is an even 
(odd) function of p: F'^(p) = Fco -I- VdP^ + • ■ ■ and 
= F^i((t • p) -I- • • •. To lowest order in momen¬ 
tum, F'^ can be taken p-independent: F^^(p) « VcoSap- 
On the other hand, F^^(p) « F„i((t„^ • p) has a p-wave 
structure and is spatially non-local. This result differs 
from the calculations of Refs. [551 and IMI which used con¬ 
stant values for both amplitudes and F'". 

Applying the SW transformation to Haim yields a 
modihed effective Kondo model: apart from the local ex¬ 
change coupling Jji, there are essentially non-local cor¬ 
rections that include p-wave couplings between conduc¬ 
tion electrons and the local impurity spin. We considered 
the simplest version in this paper and leave the more 
complex situation for a future investigation. 
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Appendix A: Self-adjoint extensions of the Dirac 
and Dimmock Hamiltonians in the half-space 

Given a linear bounded operator O, its adjoint is 
defined as {ilj\0^(j)) = {0^\(j)) for all vectors \tp) and \(j)) 
in the Hilbert space H. Moreover, O is symmetric (or 
Hermitian) if ('^jO;/)) = {O-iplf/)) or = O for all vectors 
\ip) and 1^). The set of all vectors \(j)) for which 0\cj)) is 
defined is called the domain of the operator O. For a 
bounded symmetric operator O its domain covers the 
entire space: D( 0 ) = 15(0'^) = H. 

On the other hand, if a linear operator H is unbounded 
its domain does not necessarily coincide with that of its 
adjoint. One can make these two domains coincide by 
defining them appropriately. If contains T>{H), 

and in 2 ?(iT) the two operators are the same, then we 
say that ffl is an extension of H. A symmetric operator 
H with a dense domain is self-adjoint whenever 'D(H) = 

In this section we prove that the Dirac Hamiltonian Q 
in the half-space z ^ 0 is self-adjoint in the domain of 
wavefunctions satisfying the BC ([^ (the following anal¬ 
ysis can also be seen as another derivation of this BG). 
We also determine self-adjoint extensions (SAEs) of the 
Dimmock Hamiltonian Q in the half-space. This consti¬ 
tutes a crucial step to discussing and analyzing surface 
or interface phenomena that is physically observable. 

The general theory of self-adjoint extensions can 
be found, for instance, in Ref. mi Its practi¬ 
cal applicatio n to an operator H, however, is rather 
straightforwarcPSMUIl and was made systematic by von 
Neumann’s method of deficiency indices. First, one con¬ 
structs deficiency subspaces of the adjoint operator, i.e. 
determines eigenfunctions '4’± of corresponding to 
eigenvalues ±ir] with arbitrary 77 > 0. Dimensions of 
these subspaces, the deficiency indices n±, give the num¬ 
ber of parameters needed to construct families of possible 
SAEs: if n_|_ = n_ = n = 0 the operator is already self- 
adjoint, otherwise (n > 0) its extensions need to be built. 
When n+ ^ n_ the operator cannot be made self-adjoint. 

Provided n+ = n_, the next step is to demand that 
the positive and negative deficiency subspaces be unitar- 
ily related by a n x n matrix U. This matrix is arbitrary 
and therefore the number of possible SAEs is n^. Finally, 
we require that the combination ' 0 + -|- Utp- belong to the 
domain of the original operator H. This yields BC for 
the wavefunctions that define the domain in which H is 
self-adjoint. The arbitrary unitary matrix U represents 
all possible BCs compatible with H being self-adjoint. 
One can then consider SAEs that are constrained by ad¬ 
ditional symmetry conditions, such as time-reversal in¬ 
variance or parity. 
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1. Dirac Hamiltonian in the half-space z ^ 0 


2. Dimmock Hamiltonian in the half-space 2^0 


For purely imaginary eigenvalues ±ir] of the Hamilto¬ 
nian Q it follows that Pz = = 

±iK. To hnd the corresponding eigenfunctions "0^, it is 
convenient to reduce Eq. 0 to a 2 X 2 form similar to 
Eq. 0: 


Similarly to the previous subsection, for the Dimmock 
model 0 , we have: 


2{m*vy 


1 + 


A -|-p^/2m* 


±t 1 


2A — /m*v‘^ 


^(2x2) _ 

and make a unitary transformation generated by 


^ 72 (i -i 


so that CT*) o’^)- In this representation: 


It is straightforward to check that for any values of the 
parameters A and p±, there are two normalizable solu¬ 
tions that decay with z —^ oo. Therefore, the deficiency 
indices are n_|_ = n_ = 2, and the self-adjoint extension 
of Eq. 0 is realized by a four-parametric family of BCs. 


Appendix B: Surface states in the total angular 
momentum basis 


where we dropped the unimportant, for the analysis be¬ 
low, dependence on x±^ as well as the normalization con¬ 
stant (which is the same for '0+ and '0_). Clearly, this 
solution exists for any sign of ±i/y, hence the deficiency 
indices are n_|_ = n_ = 1. By the von Neumann theorem, 
the Hamiltonian 0 has a single-parameter, n = 1, fam¬ 
ily of SAEs. The unitary matrix, connecting and '0- 
is just e^"^ with an arbitrary A. 

Possible SAEs are found in the form of BC for a general 
wavefunction p = from the domain of H. The 

condition that H is self-adjoint if 

{4>\Hp) - {H^\p) = 0 


The eigenvalue problem defined by the Dirac Hamilto¬ 
nian 0 and its BC 0 has an axial symmetry around the 
2-axis which leads to conservation of the 2-component of 
the total angular momentum = Iz + {Iz is the or¬ 
bital angular momentum). Here we will employ this sym¬ 
metry to construct surface states with a definite value of 
jz, and derive their spin structure (15) and coupling to 
the impurity [see Eq. ([01)]. 

We will work in cylindrical coordinates (r, p, z) with 
0 0 r < oo and 0 0 (/? < 27r, related to the Cartesian 
basis in Fig. 0a) via x = r cos p and y = r sin p. The 
vector product] crxpj_]2 entering the tensor spin operator 
Tz [see Eq. 0] has the form 


[cr X Pi_]z = 


0 


+ ^) 


+ ^) 

0 


[0 € and we are interested in functions such that 

'D{H) = 'D{H^)]. Substituting 0 = 0+ -|- e‘'^0_, we ob¬ 
tain (note that is equivalent to cr*'): 

Pi VTp±cosX/2 — r]smX/2 

P 2 z=o (A — uk) cos A/2 

where p is an arbitrary real constant. 

As a final step, we would like to recast this BC in a 
form B'p\z=o = 0 (detH' = 0). The matrix B' can be 
written as 

+ (1 + fe2P)o-“ - i(l - b2p)(7'^] , 

with arbitrary real 62 (notice that 61 is irrelevant). This 
BC preserves time-reversal and parity invariance of the 
Dirac Hamiltonian. The BC of Eq. 0 is recovered af¬ 
ter inverting the ^-transformation, i.e. replacing cr with 
(/3, —i/Ja^), and taking bi = 2p/(l-|-p^) and 62 = 1/p- 
Then sint? = 2p/(l -|- p^) and cos^ = (1 — P^)/(l + P^)- 


with L = —i 


^. The eigenstates of this operator are 




1 f Jm(pur)e™‘^ 
7^ lTj^+i(pir)e‘(™+i)<^ 


Here pj_ = |pj_| and Jm{x) is the Bessel function of the 
first kind, of order m. This wavefunction is analogous 
to Eq. 0 with p± replaced by a pair (p_L,m). It is 
normalized to the total surface area A\ 


d X±U^,^^,^,Upj^rnT — 


S(j)'j_-p±) 

2^ 7777 


(1 -I- t't) 


2Tr6r, 


2A 


j (2(jr'r) — 


where we used the relation between discrete and contin¬ 
uous (Dirac) ^-functions, 6pi p^ —)■ ^{p'j_p±)~^^^S{p± — 
p'j_) and —>■ 2Tr6{(j)p'^ — 4>p^) that follow 


from the vector relation S„' „, 


0PJ_ 


^(5(p_l - p'jJ = ^^(0p(^ - 0 p_l)(7j_P±) - 

P±) (see also the discussion at the beginning of 
Sec. IV). There is also a completeness relation 
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1^ y\a\Up^mT{j'-i^)\p = lJa/3<^(®_L ~ 3^ J_ ) ■ 
Using well-known properties of the Bessel function^^, we 
can relate Up^mr and plane-wave spinors of Eq. Q: 


pip±x±_ 




(Bl) 


?71= —OO 


The surface-state wavefunction (10) can be written as 
1 pp^Tix±_) = with 




<lp± 


(1 - sind)Up^rnT{r,ip)\ - 


l-sini9 \-icos‘dUp^rn-T(r,ip) 


9p_l 


It is easy to show that jzipp^mT = ijn ^)ipp^rnT- Fur¬ 
thermore, the operator from Eq. @ becomes 


H Qpir Cj*;^^,^,{-sind[(e,,-Fiej^)r'<5“o^-F 


pT 
p±mT 


+ (e. - iep)TCiO] + - r'TS^i^)}Cp 

I ^ 


2 ^ 

p'^t' 

P±r 


Qptr {-sind cr^ + SzCJ^) 




where we used the fact that Jm(0) = Smo and Opj^mr = 
i"*Cp^mT- The operators ap^mr are defined via Up^rt = 
Cpj^OT and Up^ri = tCp^It-. The latter expression differs 
from the analogous definition in Eq. (14) by a pure phase 
—i which can be tracked to the above relation between 
fermion operators Cp^mr and Cp^mr, as well as the fac¬ 
tor i™ in the expansion (Bl). When plugged into the 
Kondo Hamiltonian (13), the above expression will yield 
the model (fTbl). 


To compute spatial spin distributions we will need the 
matrix element 


Spimr(a:i,2 = 0) = ^V'p'^mr^V'p^™^ 


z^O 


-- I _T sin ^ Gm{p, p')er + Cm{p,p')ez + 

+ iT sin-dFm(p,p')e<p|, 


where p = p±r, p' = p±r', = e^cosp + By sin (p, 

Bp = —Bx sin p + By cos ip, and 



^ f Jm {p')Jra+l{p) ± Jm {p)Jm+l{p )\ 

2 y {p)Jm{p) Jm+l{p )JTn+l{p)J 


Importantly, Gm and Cm are symmetric w.r.t. in¬ 
terchange of their arguments [Gmip^p') = Gm{p',p) 
and Cm{p,p') = Cln(p^p)], while Fm is antisymmetric 
[Fm{p,p') = —Fm{p',p)]- We will only consider the case 
m = 0 and 1. By virtue of the relation J_i(p) = — Ji(p), 
Go = —Gj, Co = ~Ci and Fq = Fj, and we get 


P i ^9p'^t(1p±t r '07^/ l\ I 7T^r>^ 

Spimr -{irsm-dFo(p,p )e,^± (B2) 

± [-Tsini?Go(p,p')er + Co(p, pO^z]}, 
ith upper (lower) sign corresponding to m = 0 (1). This 
[nation reduces to (15) when p_L = p'j_ (i.e. p' = p and 
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